{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started with the Iguazio Data Science Platform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-overview\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial is designed to demonstrate some of the main capabilities of the Iguazio Data Science Platform (**\"the platform\"**).\n",
    "The tutorial walks you through a basic machine-learning (ML) workflow, which includes the following steps; before running the steps, ensure that the required [prerequisites](#gs-prerequisites) are met:\n",
    "\n",
    "1. [Create and configure a new project](#gs-step-project-create-n-config)\n",
    "2. [Ingest a data set](#gs-step-ingest-data)\n",
    "3. [Explore the data](#gs-step-explore-data)\n",
    "4. [Run model training at scale](#gs-step-run-model-training)\n",
    "5. [Test your model](#gs-step-test-model)\n",
    "6. [Serve your model](#gs-step-model-serving)\n",
    "7. [Create and run a fully automated ML pipeline](#gs-step-create-n-run-ml-pipeline)\n",
    "\n",
    "By the end of this tutorial you will learn how to\n",
    "\n",
    "1. Create a new platform project.\n",
    "2. Create functions and track experiments while running them.\n",
    "3. Store artifacts to be used and managed in a central database.\n",
    "4. Leverage the MLRun functions marketplace.\n",
    "5. Create an automated ML pipeline.\n",
    "6. Run your code on a distributed Kubernetes cluster without any DevOps overhead.\n",
    "7. Create a scalable endpoint for model serving.\n",
    "\n",
    "The tutorial uses the [Iris flower data set](http://archive.ics.uci.edu/ml/datasets/iris) &mdash; a popular data set that is widely used for basic data science tutorials.\n",
    "The data set contains a set of 150 records under five attributes &mdash; petal length, petal width, sepal length, sepal width, and species. Your model will classify the species based on the other four attributes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-prerequisites\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-mlrun-install\"></a>The tutorial uses MLRun to create a project, implement and execute an ML pipeline, and track the execution.\n",
    "(For more information about MLRun, see Step 1.)\n",
    "To use MLRun, you must first ensure that it's installed and running as a service on your platform clusters.\n",
    "Look for an `mlrun` service on the **Services** page of the platform dashboard.\n",
    "For more information and additional assistance, contact the Iguazio [support team](mailto:support@iguazio.com).\n",
    "\n",
    "To use MLRun from Jupyter Notebook, you also need to install the MLRun Python package (`mlrun`) and synchronize its version with the running MLRun service.\n",
    "This needs to be done only once per Jupyter Notebook service for the initial installation and whenever the MLRun service is updated.\n",
    "You can install and update the MLRun Python package by running the provided **align_mlrun.sh** script from Jupyter Notebook, as demonstrated in the following cell.\n",
    "For more information, see the [**weclome.ipynb**](../welcome.ipynb#mlrun-python-pkg-install-n-update) notebook or [**REAMDE.md**](../README.md#mlrun-python-pkg-install-n-update) file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!/User/align_mlrun.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-step-project-create-n-config\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Create and Configure a New Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start out by creating a new platform project and making some initial configurations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-create-project\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a New Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Projects in the platform are used to package multiple functions, workflows, and artifacts.\n",
    "Projects are created by using the `new_project` MLRun method, which receives the following parameters:\n",
    "\n",
    "- **`name`** (Required) &mdash; the project name.\n",
    "- **`context`** &mdash; the path to a local project directory (the project's context directory).\n",
    "  The project directory contains a project-configuration file (default: **project.yaml**), which defines the project, and additional generated Python code.\n",
    "  The project file is created when you save your project (using the `save` MLRun project method), as demonstrated in Step 6.\n",
    "- **`functions`** &mdash; a list of functions objects or links to function code or objects.\n",
    "- **`init_git`** &mdash; set to `True` to perform Git initialization of the project directory (`context`).\n",
    "  > **Note:** It's customary to store project code and definitions in a Git repository.\n",
    "\n",
    "Projects are visible in the MLRun dashboard only after they're saved to the MLRun database, which happens whenever you run code for a project.\n",
    "\n",
    "The following code creates a project named \"getting-started-iris-&lt;V3IO_USERNAME&gt;\", where **&lt;V3IO_USERNAME&gt;** is your current running username in the platform, and sets the project directory to a **conf** directory in the current tutorial directory (**/User/getting-started-tutorial/conf**).\n",
    "\n",
    "> **Note:** Platform projects are shared among all users of the parent tenant, to facilitate collaboration. Therefore,\n",
    ">\n",
    "> - Synchronize your projects execution with other users on your platform cluster, as needed, or use unique project names to avoid conflicts.\n",
    ">   You can easily change the default project name for this tutorial by changing the definition of the `project_name` variable in the following code.\n",
    "> - Don't include in your project proprietary information that you don't want to expose to other users.\n",
    ">   Note that while projects are a useful tool, you can easily develop and run code in the platform without using projects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project path: /User/getting-started-tutorial/conf\n",
      "Project name: getting-started-iris-iguazio\n"
     ]
    }
   ],
   "source": [
    "from os import path, getenv\n",
    "from mlrun import new_project\n",
    "\n",
    "project_name = '-'.join(filter(None, ['getting-started-iris', getenv('V3IO_USERNAME', None)])).lower()\n",
    "project_path = path.abspath('conf')\n",
    "project = new_project(project_name, project_path, init_git=True)\n",
    "\n",
    "print(f'Project path: {project_path}\\nProject name: {project_name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-mlrun-config\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure MLRun"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[MLRun](https://github.com/mlrun/mlrun) is a generic and convenient mechanism for data scientists and software developers to describe and run tasks related to machine learning in various, scalable runtime environments and ML pipelines while automatically tracking executed code, metadata, inputs, and outputs.\n",
    "MLRun integrates with the Nuclio serverless framework and with the Kubeflow Pipelines framework for running ML pipelines.\n",
    "The tutorial uses MLRun to create a project, implement and execute an ML pipeline, and track the execution.\n",
    "Before running your code, you need to set some MLRun configurations:\n",
    "\n",
    "- <a id=\"gs-mlrun-config-artifcats-path\"></a>**Artifacts path** &mdash; the location for storing versioned data artifacts (such as files, objects, data sets, and models) that are produced or consumed by functions, runs, and workflows.\n",
    "  The path can be defined either as a local directory path or as a URL (of the format `s3://*`, `v3io://*`, etc.).\n",
    "  You can set the artifacts path either by defining an `MLRUN_ARTIFACT_PATH` environment variable (which applies globally throughout the current environment) or as part of the MLRun configuration.\n",
    " \n",
    "  If the target directory doesn't exist, MLRun creates it.\n",
    "  You can use the notation `{{run.uid}}` in the path to signify the current run ID.\n",
    "  For pipelines, you can use the notation `{{workflow.uid}}` to signify the workflow ID.\n",
    "  This allows you to create a unique artifacts directory for each executed job or workflow.\n",
    "\n",
    "  After you run an MLRun job, the artifacts directory might contain one or more of the following directories:\n",
    " \n",
    "  - **plots** &mdash; a directory for storing images, figures, and plotlines.\n",
    "  - **models** &mdash; a directory for storing all trained models.\n",
    "  - **data** &mdash; a directory for storing any other type of data artifact, such as data sets.\n",
    "\n",
    "\n",
    "- <a id=\"gs-mlrun-config-mlrun-db-path\"></a>**The MLRun DB path** &mdash; the path to an MLRun database (DB) directory or the URL of an MLRun API service.\n",
    "  The MLRun service initializes `mlconf.dbpath` to the default API URL for the current environment (usually `http://mlrun-api:8080`).\n",
    "\n",
    "The following code sets the artifacts path to a **jobs** directory within the tutorial directory (**/User/getting-started-tutorial/jobs**), and the MLRun DB path to the currently configured MLRun DB path, if exists, or to `http://mlrun-api:8080` otherwise.\n",
    "(In Step 7 of the tutorial, you'll create a full pipeline and store the pipeline artifacts in a **pipeline** tutorial directory.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Artifacts path: /User/getting-started-tutorial/jobs\n",
      "MLRun DB path: http://mlrun-api:8080\n"
     ]
    }
   ],
   "source": [
    "from mlrun import run_local, NewTask, mlconf, import_function, mount_v3io\n",
    "\n",
    "# Target location for storing pipeline artifacts\n",
    "artifact_path = path.abspath('jobs')\n",
    "# MLRun DB path or API service URL\n",
    "mlconf.dbpath = mlconf.dbpath or 'http://mlrun-api:8080'\n",
    "\n",
    "print(f'Artifacts path: {artifact_path}\\nMLRun DB path: {mlconf.dbpath}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-nuclio-import-n-config\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import and Configure Nuclio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Nuclio](https://nuclio.io/) is a high-performance open-source and managed serverless framework, which is available as a predefined tenant-wide platform service (`nuclio`).\n",
    "The tutorial uses Nuclio to create and deploy serverless functions.\n",
    "Therefore, you need to import the Nuclio package and configure Nuclio for your project.\n",
    "\n",
    "The platform's Jupyter Notebook service preinstalls the [nuclio-jupyter SDK](https://github.com/nuclio/nuclio-jupyter/blob/master/README.md) for creating and deploying Nuclio functions with Python and Jupyter Notebook.\n",
    "The tutorial uses the Nuclio magic commands and annotation comments of this SDK to automate function code generation.\n",
    "The magic commands are initialized when you import the `nuclio` package.<br>\n",
    "The `%nuclio` magic commands are used to run Nuclio commands from Jupyter notebooks (`%nuclio <Nuclio command>`).\n",
    "You can also use `%%nuclio` at the start of a cell to identify the entire cell as containing Nuclio code.\n",
    "The magic commands are initialized when you import the `nuclio` package.<br>\n",
    "The `# nuclio: start-code`, `# nuclio: end-code`, and `# nuclio: ignore` section-marker annotations notify Nuclio of the beginning or end of code sections.\n",
    "Nuclio ignores all notebook code before a `# nuclio: start-code` marker or after an `# nuclio: end-code` marker.\n",
    "Nuclio translates all other notebook code sections into function code, except for sections that are marked with the `# nuclio: ignore` marker."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Nuclio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code imports the `nuclio` Python package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nuclio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Configure Nuclio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code uses the `# nuclio: start-code` marker to instruct Nuclio to start processing code only from this location, and then performs basic Nuclio function configuration &mdash; defining the name of the function's container image (`mlrun/ml-models`) and the function type (`job`).\n",
    "\n",
    "> **Note:** You can add code to define function dependencies and perform additional configuration after the `# nuclio: start-code` marker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nuclio: start-code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "%nuclio: setting spec.image to 'mlrun/ml-models'\n",
      "%nuclio: setting kind to 'job'\n"
     ]
    }
   ],
   "source": [
    "%nuclio config spec.image = \"mlrun/ml-models\"\n",
    "%nuclio config kind = \"job\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-step-ingest-data\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Ingest a Data Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define and run a Python function that downloads the Iris flower data set from Amazon Web Services (AWS) S3 bucket to the platform's local NoSQL data store, and stores it as an MLRun data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-step-ingest-data-init\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the source URL for downloading the data set.\n",
    "\n",
    "> **Note:** As explained earlier, the `nuclio: ignore` annotation is used to instruct Nuclio to ignore this cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nuclio: ignore\n",
    "# Set the source-data URL\n",
    "source_url = 'https://s3.wasabisys.com/iguazio/data/iris/iris_dataset.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-step-ingest-data-define-function\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a Data-Ingestion Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a data-ingestion function that downloads the configured data set from an AWS S3 bucket and saves it to the platform's local NoSQL data store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import path\n",
    "from mlrun.execution import MLClientCtx\n",
    "from mlrun.datastore import DataItem\n",
    "\n",
    "# Ingest a data set into the platform\n",
    "def get_data(context: MLClientCtx, source_url: DataItem, format: str = 'csv'):\n",
    "\n",
    "    iris_dataset = source_url.as_df()\n",
    "\n",
    "    target_path = path.join(context.artifact_path, 'data')\n",
    "    # Optionally print data to your logger\n",
    "    context.logger.info('Saving Iris data set to {} ...'.format(target_path))\n",
    "\n",
    "    # Store the data set in your artifacts database\n",
    "    context.log_dataset('iris_dataset', df=iris_dataset, format=format,\n",
    "                        index=False, artifact_path=target_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell uses the `# nuclio: end-code` marker to mark the end of a Nuclio code section and instruct Nuclio to stop parsing the notebook at this point.<br>\n",
    "> **IMPORTANT:** Do not remove the end-code cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nuclio: end-code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='gs-run-ingest-func'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the Data-Ingestion Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MLRun enables you to run function code either locally in your preferred development environment (IDE) or web notebook (in this case, Jupyter Notebook), or on a large cluster (in this case, the platform's distributed Kubernetes cluster).\n",
    "\n",
    "For either execution method, you can specify function parameters, inputs, and outputs either by passing them as parameters to the run method (as done in this tutorial) or by configuring them in a task object that is then passed to the run method; (for more information about tasks, see Step 4 of this tutorial).\n",
    "In each run, you can specify new function inputs, parameters/hyperparameters, etc.\n",
    "For more information, see the [**mlrun_basics**](https://github.com/mlrun/mlrun/blob/master/examples/mlrun_basics.ipynb) example notebook.\n",
    "\n",
    "After the code execution completes, review the output, as demonstrated in the next sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the Function Locally in Jupyter Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start out by running your function locally from within Jupyter Notebook.\n",
    "This is done by using the MLRun `run_local` method.\n",
    "\n",
    "When using this method, the function's inputs and outputs are automatically recorded and stored in the MLRun experiment and data-tracking database (**\"the MLRun DB\"**)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2020-10-15 08:03:40,206 [info] starting run get_data uid=c1c474e7a43f480aabcadf7d9a19d2bb  -> http://mlrun-api:8080\n",
      "> 2020-10-15 08:03:41,152 [info] Saving Iris data set to /User/getting-started-tutorial/jobs/data ...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #b3edff;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #ffe6cc;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #ffe6cc;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>getting-started-iris-iguazio</td>\n",
       "      <td><div title=\"c1c474e7a43f480aabcadf7d9a19d2bb\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/c1c474e7a43f480aabcadf7d9a19d2bb/info\" target=\"_blank\" >...9a19d2bb</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 15 08:03:40</td>\n",
       "      <td>completed</td>\n",
       "      <td>get_data</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">kind=handler</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">host=jupyter-64f64ccc5c-fs5q5</div></td>\n",
       "      <td><div title=\"https://s3.wasabisys.com/iguazio/data/iris/iris_dataset.csv\">source_url</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result505435e7\" title=\"/files/getting-started-tutorial/jobs/data/iris_dataset.csv\">iris_dataset</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result505435e7-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result505435e7-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result505435e7\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result505435e7-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to track results use .show() or .logs() or in CLI: \n",
      "!mlrun get run c1c474e7a43f480aabcadf7d9a19d2bb --project getting-started-iris-iguazio , !mlrun logs c1c474e7a43f480aabcadf7d9a19d2bb --project getting-started-iris-iguazio\n",
      "> 2020-10-15 08:03:41,303 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "# Run the data-ingestion function locally in Jupyter Notebook\n",
    "get_data_run = run_local(name='get_data',\n",
    "                         handler=get_data,\n",
    "                         inputs={'source_url': source_url},\n",
    "                         project=project_name, artifact_path=artifact_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-ingest-func-output-review\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review the Run Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output cell for your function execution contains a table with run information &mdash; including the state of the execution, all inputs and parameters, and the execution results and artifacts.\n",
    "Click on the `iris_dataset` artifact in the **artifacts** column to see a short summary of the data set, as illustrated in the following image:\n",
    "<br><br>\n",
    "![getting_started-get_data-details](./assets/getting_started-get_data-details.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MLRun features a graphical user interface (**\"the MLRun dashboard\"**), which displays detailed information about executed jobs, artifacts, and functions &mdash; including job states, inputs, parameters, artifacts, execution results, and logs.\n",
    "This dashboard also allows you to filter the display based on different criteria.<br>\n",
    "To open the dashboard, simply select the MLRun service link (`mlrun`) from the **Services** page of the platform dashboard.\n",
    "\n",
    "On the MLRun dashboard's **Projects** home page, select the **Jobs** link for the current project (\"getting-started-iris\") to open a **Jobs** page with a **Monitor** tab that displays information about the project's executed jobs:\n",
    "<br><br>\n",
    "<img src=\"./assets/getting_started-ui-jobs-monitor.png\" alt=\"ui-jobs-monitor\" width=\"800\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select the `get_data` function to see detailed information about the function and its execution &mdash; including the execution start time and status, metadata, inputs, artifacts, run results, and logs:\n",
    "<br><br>\n",
    "<img src=\"./assets/getting_started-get_data-ui-jobs.png\" alt=\"get-iris-data-ui-jobs\" width=\"800\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Return to the dashboard's **Projects** home page and select the **Artifacts** link for the current project to open an **Artifacts** page with a **Monitor** tab that displays information about your project's artifacts.\n",
    "Select the \"get_data_iris_dataset\" artifact to view information about the \"iris_dataset\" artifact of the `get_data` function (**iris_dataset.csv**); note that you can also preview the data set and see related metadata:\n",
    "<br><br>\n",
    "<img src=\"./assets/getting_started-get_data-ui-artifacts.png\" alt=\"get-iris-data-ui-artifacts\" width=\"800\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note:** There are also other ways that you can view run artifacts, in addition to the notebook run-output table and the MLRun dashboard:\n",
    "> - In your notebook, use Python print or display methods, such as `IPython.display`, to display the artifact of a run object, as demonstrated later in this tutorial.\n",
    "> - On the platform dashboard, select **Data** from the side navigation menu, and then select your data container from the **Browse** tab; the tutorial stores data in the \"users\" container.\n",
    ">   Then, navigate to your project's artifacts directory, locate the artifact file, and double-click it to view it in the browse table.<br>\n",
    ">   You can also download platform data files from the dashboard to your local environment.\n",
    "> - In the JupyterLab file browser, navigate to the project's artifacts directory (**/User/getting-started-tutorial/jobs** for this tutorial), locate the artifact file, and select it to view the file from the Jupyter UI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the Function on a Cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the key benefits of Iguazio Data Science Platform is the ability to easily run any code on a distributed Kubernetes cluster and achieve better performance.\n",
    "To do so, you first need to convert your notebook code into a function that can run on the cluster, and then run this function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Convert the Notebook Code Into a Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the MLRun `code_to_function` method to convert your notebook code into a project MLRun function &mdash; a function object with embedded code, which can run on the cluster.\n",
    "This will also come in handy later in the tutorial, after you create an automated pipeline.\n",
    "\n",
    "The following code converts the code of your local `get-data` data-ingestion function into a `gen_func` project function.\n",
    "(For more information about the `set_function` MLRun project method, which is used to add the new project function, see Step 3.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f635cde1fd0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from mlrun import code_to_function\n",
    "\n",
    "# Convert the local get_data function into a gen_func project function\n",
    "gen_func = code_to_function(name='get-data')\n",
    "project.set_function(gen_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Run the Function on the Cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you create the project function, you can run it on the platform's distributed Kubernetes cluster.\n",
    "The execution code is similar to the code used for the local execution, except that the `run_local` method is replaced with the `run` method.\n",
    "\n",
    "When the execution is done, review the output as you did for the local execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2020-10-15 08:03:44,500 [info] starting run get_data uid=b98a44ab835d4296803c1af70720e266  -> http://mlrun-api:8080\n",
      "> 2020-10-15 08:03:44,617 [info] Job is running in the background, pod: get-data-8mjmd\n",
      "> 2020-10-15 08:03:53,423 [info] Saving Iris data set to /User/getting-started-tutorial/jobs/data ...\n",
      "> 2020-10-15 08:03:53,536 [info] run executed, status=completed\n",
      "final state: succeeded\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #b3edff;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #ffe6cc;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #ffe6cc;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>getting-started-iris-iguazio</td>\n",
       "      <td><div title=\"b98a44ab835d4296803c1af70720e266\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/b98a44ab835d4296803c1af70720e266/info\" target=\"_blank\" >...0720e266</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 15 08:03:52</td>\n",
       "      <td>completed</td>\n",
       "      <td>get_data</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">host=get-data-8mjmd</div></td>\n",
       "      <td><div title=\"https://s3.wasabisys.com/iguazio/data/iris/iris_dataset.csv\">source_url</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulta8748f95\" title=\"/files/getting-started-tutorial/jobs/data/iris_dataset.csv\">iris_dataset</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resulta8748f95-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resulta8748f95-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resulta8748f95\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resulta8748f95-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to track results use .show() or .logs() or in CLI: \n",
      "!mlrun get run b98a44ab835d4296803c1af70720e266 --project getting-started-iris-iguazio , !mlrun logs b98a44ab835d4296803c1af70720e266 --project getting-started-iris-iguazio\n",
      "> 2020-10-15 08:03:56,867 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "get_data_func = project.func('get-data').apply(mount_v3io())\n",
    "get_data_run = get_data_func.run(name='get_data',\n",
    "                                 handler='get_data',\n",
    "                                 inputs={'source_url': source_url},\n",
    "                                 artifact_path=artifact_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-step-explore-data\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Explore the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data exploration makes up a big part of the data science work.\n",
    "This step demonstrates how you can use pandas DataFrames or an MLRun marketplace function to read your data set, analyze it to extract statistics, and visualize the information to gain more insight into your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-explore-data-w-pandas-df\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore the Data with Pandas DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following code to use pandas DataFrames to read your data set, extract some basic statistics, and display them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read your data set\n",
    "from mlrun.run import get_dataitem\n",
    "df = get_dataitem(get_data_run.outputs['iris_dataset']).as_df()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <th>petal length (cm)</th>\n",
       "      <th>petal width (cm)</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)  \\\n",
       "0                5.1               3.5                1.4               0.2   \n",
       "1                4.9               3.0                1.4               0.2   \n",
       "2                4.7               3.2                1.3               0.2   \n",
       "3                4.6               3.1                1.5               0.2   \n",
       "4                5.0               3.6                1.4               0.2   \n",
       "\n",
       "   label  \n",
       "0      0  \n",
       "1      0  \n",
       "2      0  \n",
       "3      0  \n",
       "4      0  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display a portion of the read data\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of rows: 150\n"
     ]
    }
   ],
   "source": [
    "# Calculate and display the number of data-set items\n",
    "print(f'Total number of rows: {len(df)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"8\" halign=\"left\">sepal length (cm)</th>\n",
       "      <th colspan=\"2\" halign=\"left\">sepal width (cm)</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"2\" halign=\"left\">petal length (cm)</th>\n",
       "      <th colspan=\"8\" halign=\"left\">petal width (cm)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>...</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>label</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>50.0</td>\n",
       "      <td>5.006</td>\n",
       "      <td>0.352490</td>\n",
       "      <td>4.3</td>\n",
       "      <td>4.800</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>5.8</td>\n",
       "      <td>50.0</td>\n",
       "      <td>3.428</td>\n",
       "      <td>...</td>\n",
       "      <td>1.575</td>\n",
       "      <td>1.9</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.246</td>\n",
       "      <td>0.105386</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50.0</td>\n",
       "      <td>5.936</td>\n",
       "      <td>0.516171</td>\n",
       "      <td>4.9</td>\n",
       "      <td>5.600</td>\n",
       "      <td>5.9</td>\n",
       "      <td>6.3</td>\n",
       "      <td>7.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>2.770</td>\n",
       "      <td>...</td>\n",
       "      <td>4.600</td>\n",
       "      <td>5.1</td>\n",
       "      <td>50.0</td>\n",
       "      <td>1.326</td>\n",
       "      <td>0.197753</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>1.5</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>50.0</td>\n",
       "      <td>6.588</td>\n",
       "      <td>0.635880</td>\n",
       "      <td>4.9</td>\n",
       "      <td>6.225</td>\n",
       "      <td>6.5</td>\n",
       "      <td>6.9</td>\n",
       "      <td>7.9</td>\n",
       "      <td>50.0</td>\n",
       "      <td>2.974</td>\n",
       "      <td>...</td>\n",
       "      <td>5.875</td>\n",
       "      <td>6.9</td>\n",
       "      <td>50.0</td>\n",
       "      <td>2.026</td>\n",
       "      <td>0.274650</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.8</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      sepal length (cm)                                              \\\n",
       "                  count   mean       std  min    25%  50%  75%  max   \n",
       "label                                                                 \n",
       "0                  50.0  5.006  0.352490  4.3  4.800  5.0  5.2  5.8   \n",
       "1                  50.0  5.936  0.516171  4.9  5.600  5.9  6.3  7.0   \n",
       "2                  50.0  6.588  0.635880  4.9  6.225  6.5  6.9  7.9   \n",
       "\n",
       "      sepal width (cm)         ... petal length (cm)      petal width (cm)  \\\n",
       "                 count   mean  ...               75%  max            count   \n",
       "label                          ...                                           \n",
       "0                 50.0  3.428  ...             1.575  1.9             50.0   \n",
       "1                 50.0  2.770  ...             4.600  5.1             50.0   \n",
       "2                 50.0  2.974  ...             5.875  6.9             50.0   \n",
       "\n",
       "                                                 \n",
       "        mean       std  min  25%  50%  75%  max  \n",
       "label                                            \n",
       "0      0.246  0.105386  0.1  0.2  0.2  0.3  0.6  \n",
       "1      1.326  0.197753  1.0  1.2  1.3  1.5  1.8  \n",
       "2      2.026  0.274650  1.4  1.8  2.0  2.3  2.5  \n",
       "\n",
       "[3 rows x 32 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display statistics grouped by label\n",
    "df.groupby(['label']).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-explore-data-w-mlrun-func\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore the Data with an MLRun Marketplace Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can perform further data exploration by leveraging **the MLRun functions marketplace** (a.k.a. \"the MLRun functions hub\").\n",
    "This marketplace is a centralized location for open-source contributions of function components that are commonly used in machine-learning development.\n",
    "The location of the marketplace is configured via the `hub_url` MLRun configuration.\n",
    "By default, it points to the [mlrun/functions](https://github.com/mlrun/functions/) GitHub repository.\n",
    "\n",
    "In this step you'll use the [`describe` marketplace function](https://github.com/mlrun/functions/tree/master/describe), which performs data exploration on a provided data set, to extract information from your data set, analyze it, and visualize relevant information in different ways."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add an Exploration Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `set_function` MLRun project method, which adds or updates a function object in a project, to load the `describe` marketplace function into a new `describe` project function. \n",
    "The tutorial code sets the first two `set_function` method parameters:\n",
    "\n",
    "- **`func`** &mdash; identities the function to load.\n",
    "  This parameter can be set to a function object or to the URL of a function specification or code file.\n",
    "  MLRun supports multiple types of URL formats.\n",
    "  The example uses the `hub://...` format to point to the `describe` function-code directory in the MLRun functions marketplace (`'hub://describe'`).\n",
    "- **`name`** &mdash; the name of the project function that is being set.\n",
    "  The example names the project function `describe` (like the name of the marketplace function, although this isn't required).\n",
    "\n",
    "After you add the function project, create a `describe` function object by using `apply(mount_v3io())` to attach an Iguazio Data Science Platform data volume (a.k.a. \"a v3io volume\") to your `describe` project function.\n",
    "This connects your function to the platform's shared file system and allows you to pass data from the platform to the project function and get back the results (plots) directly into your notebook.\n",
    "You can also select to use other supported mount methods &mdash; for example, for using NFS or object storage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "describe = project.set_function('hub://describe', 'describe').apply(mount_v3io())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### View the Function Documentation\n",
    "\n",
    "Use the `doc` method to view the embedded documentation of the `describe` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function: describe\n",
      "describe and visualizes dataset stats\n",
      "default handler: summarize\n",
      "entry points:\n",
      "  summarize: Summarize a table\n",
      "    context(MLClientCtx)  - the function context, default=\n",
      "    table(DataItem)  - MLRun input pointing to pandas dataframe (csv/parquet file path), default=\n",
      "    label_column(str)  - ground truth column label, default=None\n",
      "    class_labels(List[str])  - label for each class in tables and plots, default=[]\n",
      "    plot_hist(bool)  - (True) set this to False for large tables, default=True\n",
      "    plots_dest(str)  - destination folder of summary plots (relative to artifact_path), default=plots\n",
      "    update_dataset  - when the table is a registered dataset update the charts in-place, default=False\n"
     ]
    }
   ],
   "source": [
    "describe.doc()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the Exploration Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following code to execute the `describe` project function as a Kubernetes job by using the MLRun `run` method.\n",
    "The returned run object is stored in a `describe_run` variable.\n",
    "\n",
    "The location of the data set is the only input that you need to provide.\n",
    "This information is provided as a `table` input artifact that points to the `iris_dataset` output artifact of the `get_data` job that you ran in the previous steps (using the job's `get_data_run` run object &mdash; see Step 2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2020-10-15 08:03:57,693 [info] starting run describe-summarize uid=b13e7f63dc2744f9beb8b2360bcc760f  -> http://mlrun-api:8080\n",
      "> 2020-10-15 08:03:57,824 [info] Job is running in the background, pod: describe-summarize-42x6q\n",
      "> 2020-10-15 08:04:10,042 [info] run executed, status=completed\n",
      "final state: succeeded\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #b3edff;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #ffe6cc;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #ffe6cc;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>getting-started-iris-iguazio</td>\n",
       "      <td><div title=\"b13e7f63dc2744f9beb8b2360bcc760f\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/b13e7f63dc2744f9beb8b2360bcc760f/info\" target=\"_blank\" >...0bcc760f</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 15 08:04:04</td>\n",
       "      <td>completed</td>\n",
       "      <td>describe-summarize</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">host=describe-summarize-42x6q</div></td>\n",
       "      <td><div title=\"store://getting-started-iris-iguazio/get_data_iris_dataset#b98a44ab835d4296803c1af70720e266\">table</div></td>\n",
       "      <td><div class=\"dictlist\">label_column=label</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultae5da379\" title=\"/files/getting-started-tutorial/jobs/plots/hist.html\">histograms</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultae5da379\" title=\"/files/getting-started-tutorial/jobs/plots/violin.html\">violin</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultae5da379\" title=\"/files/getting-started-tutorial/jobs/plots/imbalance.html\">imbalance</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultae5da379\" title=\"/files/getting-started-tutorial/jobs/plots/imbalance-weights-vec.csv\">imbalance-weights-vec</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultae5da379\" title=\"/files/getting-started-tutorial/jobs/plots/correlation-matrix.csv\">correlation-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultae5da379\" title=\"/files/getting-started-tutorial/jobs/plots/corr.html\">correlation</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultae5da379-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultae5da379-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultae5da379\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultae5da379-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to track results use .show() or .logs() or in CLI: \n",
      "!mlrun get run b13e7f63dc2744f9beb8b2360bcc760f --project getting-started-iris-iguazio , !mlrun logs b13e7f63dc2744f9beb8b2360bcc760f --project getting-started-iris-iguazio\n",
      "> 2020-10-15 08:04:17,084 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "describe_run = describe.run(params={'label_column': 'label'},\n",
    "                            inputs={\"table\":\n",
    "                                    get_data_run.outputs['iris_dataset']},\n",
    "                            artifact_path=artifact_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Review the Run Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output cell for your code execution contains a run-information table.\n",
    "You can also view run information in the MLRun dashboard; see the output-review information in Step 2, only this time look for the `describe-summarize` job and related artifacts.\n",
    "\n",
    "The `describe` function generates three HTML output artifacts, which provide visual insights for your data set &mdash; `histograms`, `imbalance`, and `correlation`.\n",
    "The artifacts are stored as HTML files in your project's artifacts directory, under **/User/getting-started-tutorial/jobs/plots/**.\n",
    "The following code displays the artifact files in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h3>Histograms</h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3 style=\"text-align:center\">histograms</h3>\n",
       "<img title=\"histograms\" src=\"\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, HTML\n",
    "\n",
    "# Display the `histograms` artifact\n",
    "display(HTML('<h3>Histograms</h3>'), HTML(describe_run.outputs['histograms']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h3>Imbalance<h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3 style=\"text-align:center\">imbalance</h3>\n",
       "<img title=\"imbalance\" src=\"\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display the `imbalance` artifact\n",
    "display(HTML('<h3>Imbalance<h3>'),\n",
    "        HTML(filename=describe_run.outputs['imbalance']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h3>Correlation Matrix<h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3 style=\"text-align:center\">Correlation Matrix</h3>\n",
       "<img title=\"correlation\" src=\"\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display the `correlation` artifact\n",
    "display(HTML('<h3>Correlation Matrix<h3>'),\n",
    "        HTML(filename=describe_run.outputs['correlation']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-step-run-model-training\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Run Model Training at Scale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's time to train your model.\n",
    "Start by adding a training project function that uses the [`sklearn_classifier` MLRun marketplace function](https://github.com/mlrun/functions/tree/master/sklearn_classifier) to train three scikit-learn models; (you can use this function to train any scikit-learn model).\n",
    "Then, use an MLRun task to run your function in order to train and evaluate the models and select the optimal model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add a Training Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following code to add to your project a `train` function that uses the `sklearn_classifier` marketplace function code, and create a related `train` function object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = project.set_function('hub://sklearn_classifier', 'train').apply(mount_v3io())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run a Model-Training Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code uses the MLRun `NewTask` method to create a new `train` MLRun task and a related `train_task` task object for executing the `train` project function.\n",
    "An MLRun **task** defines the parameters, inputs, and outputs of a logical job or task to execute.\n",
    "The advantage of tasks is that they're independent of the runtime environment.\n",
    "Hence, you can define a task once and use it in different scenarios.\n",
    "\n",
    "The `NewTask` `inputs` parameter is set to the `iris_dataset` data set, as stored in the output artifacts of the previously executed `get_data` job.\n",
    "The `test_size` training parameter (`params.test_size`) is set to 0.10.\n",
    "This means that the `iris_dataset` data set will make up 90% of the training data set, and the remaining 10% will be the scikit-learn test data set.\n",
    "\n",
    "The `train_task` task (and therefore the `train` function) is executed by using the `run` method.\n",
    "The returned run object is stored in a `train_run` variable.\n",
    "\n",
    "The code uses the `with_hyper_params` task method, which enables running multiple iterations of the same task with different parameter combinations, comparing the results, and selecting the optimal combination based on the specified selection criteria.\n",
    "In this example, there's a single `models` parameter with different values (model names) so as to compare multiple models; the model with the most accurate training results is selected as the optimal model.\n",
    "The `with_hyper_params` method receives the following parameters\n",
    "\n",
    "- **`hyperparams`** &mdash; a dictionary of parameter names and parameter-values lists.\n",
    "  The example defines a single hyperparameter named `model_pkg_class`, and sets its value to a `models` variable that contains a list of three scikit-learn models to train and evaluate. \n",
    "- **`selector`** &mdash; a string of the format `'<min/max>.<validation criterion>'`, which defines a selection criterion for selecting the winning run iteration.\n",
    "  For example, `\"max.accuracy\"` selects the iteration with the most accurate results while `\"min.loss\"` selects the iteration with minimum loss.\n",
    "  The example uses the `max.accuracy` selector to select the model with the most accurate training results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure the models to train\n",
    "models = [\"sklearn.ensemble.RandomForestClassifier\", \n",
    "          \"sklearn.linear_model.LogisticRegression\",\n",
    "          \"sklearn.ensemble.AdaBoostClassifier\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2020-10-15 08:04:17,667 [info] starting run train uid=cea10bf1c2bd440f9da605ec3c804dde  -> http://mlrun-api:8080\n",
      "> 2020-10-15 08:04:17,811 [info] Job is running in the background, pod: train-qh6md\n",
      "> 2020-10-15 08:04:29,487 [info] best iteration=2, used criteria max.accuracy\n",
      "> 2020-10-15 08:04:29,709 [info] run executed, status=completed\n",
      "/opt/conda/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:764: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n",
      "final state: succeeded\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #b3edff;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #ffe6cc;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #ffe6cc;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>default</td>\n",
       "      <td><div title=\"cea10bf1c2bd440f9da605ec3c804dde\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/default/jobs/cea10bf1c2bd440f9da605ec3c804dde/info\" target=\"_blank\" >...3c804dde</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 15 08:04:25</td>\n",
       "      <td>completed</td>\n",
       "      <td>train</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=iguazio</div></td>\n",
       "      <td><div title=\"store://getting-started-iris-iguazio/get_data_iris_dataset#b98a44ab835d4296803c1af70720e266\">dataset</div></td>\n",
       "      <td><div class=\"dictlist\">sample=-1</div><div class=\"dictlist\">label_column=label</div><div class=\"dictlist\">test_size=0.1</div></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=2</div><div class=\"dictlist\">accuracy=0.9705882352941176</div><div class=\"dictlist\">test-error=0.029411764705882353</div><div class=\"dictlist\">auc-micro=0.9961072664359862</div><div class=\"dictlist\">auc-weighted=0.9949732620320856</div><div class=\"dictlist\">f1-score=0.9679633867276888</div><div class=\"dictlist\">precision_score=0.9666666666666667</div><div class=\"dictlist\">recall_score=0.9722222222222222</div></td>\n",
       "      <td><div title=\"/User/getting-started-tutorial/jobs/data/2/test_set.parquet\">test_set</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result5107d92a\" title=\"/files/getting-started-tutorial/jobs/model/plots/2/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result5107d92a\" title=\"/files/getting-started-tutorial/jobs/model/plots/2/precision-recall-multiclass.html\">precision-recall-multiclass</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result5107d92a\" title=\"/files/getting-started-tutorial/jobs/model/plots/2/roc-multiclass.html\">roc-multiclass</div><div title=\"/User/getting-started-tutorial/jobs/model/2\">model</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result5107d92a\" title=\"/files/getting-started-tutorial/jobs/iteration_results.csv\">iteration_results</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result5107d92a-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result5107d92a-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result5107d92a\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result5107d92a-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to track results use .show() or .logs() or in CLI: \n",
      "!mlrun get run cea10bf1c2bd440f9da605ec3c804dde --project default , !mlrun logs cea10bf1c2bd440f9da605ec3c804dde --project default\n",
      "> 2020-10-15 08:04:37,055 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "# Create a training task\n",
    "train_task = NewTask(name=\"train\",\n",
    "                     params={\"sample\": -1,\n",
    "                             \"label_column\": \"label\",\n",
    "                             \"test_size\": 0.10},\n",
    "                     inputs={\"dataset\": get_data_run.outputs['iris_dataset']})\n",
    "\n",
    "# Run the training task\n",
    "train_run = train.run(train_task.with_hyper_params({'model_pkg_class': models},\n",
    "                                                   selector='max.accuracy'),\n",
    "                      artifact_path=artifact_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-train-model-run-output-review\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review the Run Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with the previous executions, you can view extensive run information and artifacts from Jupyter Notebook and the MLRun dashboard, as well as browse the task artifacts from the platform dashboard.\n",
    "See the output-review information in the previous steps.\n",
    "\n",
    "Your project's artifacts directory (**/User/getting-started-tutorial/jobs**) contains an **iteration_results.csv** output artifact with the results for the training-task run iterations, and **&lt;iteration number&gt;** subdirectories with the artifacts for all the run iterations (i.e., for each evaluated model).<br>\n",
    "The **plots** subdirectory has HTML output artifacts for the selected run iteration; (the **tests** subdirectory contains the artifacts for the test data set).\n",
    "\n",
    "Use the following code to extract and display information from the run object &mdash; the name of the selected trained model, the best accuracy that was achieved with this model, and the `confusion` and `roc` HTML output artifacts for the optimal run iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Best model:</b> sklearn.linear_model.LogisticRegression"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<b>Accuracy:</b> 0.9705882352941176"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3>Confusion Matrix:</h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3 style=\"text-align:center\">Confusion Matrix - Normalized Plot</h3>\n",
       "<img title=\"confusion-matrix\" src=\"\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3>ROC Curve:</h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3 style=\"text-align:center\">Multiclass ROC Curve</h3>\n",
       "<img title=\"roc-multiclass\" src=\"\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display the name of the selected model\n",
    "# best_iteration is 1 based, get the best model from the model list by substracting 1 \n",
    "display(HTML(f'<b>Best model:</b> '\n",
    "             f'{models[train_run.outputs[\"best_iteration\"] - 1]}'))\n",
    "\n",
    "# Display the accuracy for the optimal run iteration\n",
    "display(HTML(f'<b>Accuracy:</b> {train_run.outputs[\"accuracy\"]}'))\n",
    "\n",
    "# Display HTML output artifacts\n",
    "display(HTML('<h3>Confusion Matrix:</h3>'),\n",
    "        HTML(filename=train_run.outputs['confusion-matrix']))\n",
    "display(HTML('<h3>ROC Curve:</h3>'), HTML(filename=train_run.outputs['roc-multiclass']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-step-test-model\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Test Your Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have a trained model, you can test it: run a task that uses the [`test_classifier` marketplace function](https://github.com/mlrun/functions/tree/master/test_classifier) to run the selected trained model against the test data set, as returned for the training task (`train`) in the previous step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add a Test Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following code to add to your project a `test` function that uses the `test_classifier` marketplace function code, and create a related `test` function object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = project.set_function('hub://test_classifier', 'test').apply(mount_v3io())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run a Model-Testing Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use a task to run the model-testing function:\n",
    "\n",
    "- Create a new `test` task and a related `test_task` task object for executing the `test` project function, which runs the `test_classifier` marketplace function.<br>\n",
    "  In the `NewTask` call, configure parameters for the test function (`params`), and provide the selected trained model from the `train` task as an input artifact (`inputs`).\n",
    "- Run the task by using the `run` MLRun method, and assign the returned run object to a `test_run` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2020-10-15 08:04:37,635 [info] starting run test uid=26acde80f60d44b7bb06d7c8ff62637b  -> http://mlrun-api:8080\n",
      "> 2020-10-15 08:04:37,770 [info] Job is running in the background, pod: test-snxnw\n",
      "> 2020-10-15 08:04:47,129 [info] run executed, status=completed\n",
      "final state: succeeded\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #b3edff;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #ffe6cc;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #ffe6cc;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>default</td>\n",
       "      <td><div title=\"26acde80f60d44b7bb06d7c8ff62637b\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/default/jobs/26acde80f60d44b7bb06d7c8ff62637b/info\" target=\"_blank\" >...ff62637b</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 15 08:04:46</td>\n",
       "      <td>completed</td>\n",
       "      <td>test</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">host=test-snxnw</div></td>\n",
       "      <td><div title=\"store://default/train_model#cea10bf1c2bd440f9da605ec3c804dde\">models_path</div><div title=\"store://default/train_test_set#cea10bf1c2bd440f9da605ec3c804dde\">test_set</div></td>\n",
       "      <td><div class=\"dictlist\">label_column=label</div><div class=\"dictlist\">plots_dest=plots/test</div></td>\n",
       "      <td><div class=\"dictlist\">accuracy=0.9333333333333333</div><div class=\"dictlist\">test-error=0.06666666666666667</div><div class=\"dictlist\">auc-micro=0.9977777777777778</div><div class=\"dictlist\">auc-weighted=1.0</div><div class=\"dictlist\">f1-score=0.9137254901960784</div><div class=\"dictlist\">precision_score=0.8888888888888888</div><div class=\"dictlist\">recall_score=0.9629629629629629</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultbe655133\" title=\"/files/getting-started-tutorial/jobs/plots/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultbe655133\" title=\"/files/getting-started-tutorial/jobs/plots/precision-recall-multiclass.html\">precision-recall-multiclass</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultbe655133\" title=\"/files/getting-started-tutorial/jobs/plots/roc-multiclass.html\">roc-multiclass</div><div title=\"/User/getting-started-tutorial/jobs/test_set_preds.parquet\">test_set_preds</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultbe655133-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultbe655133-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultbe655133\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultbe655133-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to track results use .show() or .logs() or in CLI: \n",
      "!mlrun get run 26acde80f60d44b7bb06d7c8ff62637b --project default , !mlrun logs 26acde80f60d44b7bb06d7c8ff62637b --project default\n",
      "> 2020-10-15 08:04:57,005 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "test_task = NewTask(name=\"test\",\n",
    "                    params={\"label_column\": \"label\",\n",
    "                            \"plots_dest\": path.join(\"plots\", \"test\")},\n",
    "                    inputs={\"models_path\": train_run.outputs['model'],\n",
    "                            \"test_set\": train_run.outputs['test_set']}\n",
    "                    )\n",
    "test_run = test.run(test_task,\n",
    "                    artifact_path=artifact_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-test-model-run-output-review\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review the Run Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the output information for your run in Jupyter Notebook and on the MLRun dashboard.\n",
    "\n",
    "Use the following code to display information from the run object &mdash; the accuracy of the model, and the `confusion` and `roc` HTML output artifacts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Test Accuracy:</b> 0.9333333333333333"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3>Confusion matrix:</h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3 style=\"text-align:center\">Confusion Matrix - Normalized Plot</h3>\n",
       "<img title=\"confusion-matrix\" src=\"\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3>ROC Curve:</h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3 style=\"text-align:center\">Multiclass ROC Curve</h3>\n",
       "<img title=\"roc-multiclass\" src=\"\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display the model accuracy\n",
    "display(HTML(f'<b>Test Accuracy:</b> {test_run.outputs[\"accuracy\"]}'))\n",
    "\n",
    "# Display HTML output artifacts\n",
    "display(HTML(\"<h3>Confusion matrix:</h3>\"),\n",
    "        HTML(filename=test_run.outputs['confusion-matrix']))\n",
    "display(HTML(\"<h3>ROC Curve:</h3>\"), HTML(filename=test_run.outputs['roc-multiclass']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-step-model-serving\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Serve Your Model\n",
    "\n",
    "Add to your project a real-time model-server function (`serving`) via the [`model_server`](https://github.com/mlrun/functions/tree/master/model_server) MLRun marketplace function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "serving = project.set_function('hub://model_server', 'serving').apply(mount_v3io())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-deploy-serving-function\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the Serving Function\n",
    "\n",
    "Set the trained model and deploy the serving function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = 'iris_dataset_v1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2020-10-15 08:04:57,461 [info] deploy started\n",
      "[nuclio] 2020-10-15 08:04:57,523 project name not found created new (getting-started-iris-iguazio)\n",
      "[nuclio] 2020-10-15 08:04:59,812 (info) Building processor image\n",
      "[nuclio] 2020-10-15 08:05:00,828 (info) Build complete\n",
      "[nuclio] 2020-10-15 08:05:10,985 (info) Function deploy complete\n",
      "[nuclio] 2020-10-15 08:05:10,999 done creating getting-started-iris-iguazio-model-server, function address: 192.168.224.70:31378\n"
     ]
    }
   ],
   "source": [
    "serving.add_model(model_name, train_run.outputs['model'])\n",
    "serving_addr = serving.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now run the model-serving function.\n",
    "MLRun leverages Nuclio on Kubernetes: Nuclio scales up and down automatically and scales to zero when unused.\n",
    "Therefore, unlike virtual machines (VMs) that consume resources and require you to delete the endpoint when unused, here you don't need to worry about deleting this serving function because of resource costs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-test-model-from-notebook\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the Model from the Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, define a test inputs vector (see `test_data`), and send the inputs to the serving function's `/predict` endpoint (see `predict_url`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = [5.1, 3.5, 1.4, 0.2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "def data_to_serving_input(data):\n",
    "    return json.dumps({\"instances\": [data]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_url = f'{serving_addr}/{model_name}/predict'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "resp = requests.put(predict_url, json=data_to_serving_input(test_data))\n",
    "print(resp.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-test-model-via-tester-function\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the Model Using A Model-Tester Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A much more robust way to test your serving function is by creating a test function that sends a test data set and compares the output of the model-serving function to the expected output.\n",
    "To do this, add to your project a REST model-tester function (`serving-tester`) that runs the model-server function with a specific table via the [`model_server_tester` MLRun marketplace function](https://github.com/mlrun/functions/tree/master/model_server_tester), which issues HTTP calls to the model server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "serving_tester = project.set_function('hub://model_server_tester', 'serving-tester').apply(mount_v3io())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the serving tester, with the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2020-10-15 08:05:11,677 [info] starting run serving-tester uid=27133308b9d944ad93588934c9f34284  -> http://mlrun-api:8080\n",
      "> 2020-10-15 08:05:11,804 [info] Job is running in the background, pod: serving-tester-t4z5v\n",
      "> 2020-10-15 08:05:17,337 [info] testing with dataset against http://192.168.224.70:31378, model: iris_dataset_v1\n",
      "> 2020-10-15 08:05:17,530 [info] run 15 tests, 0 errors and 14 match expected value\n",
      "> 2020-10-15 08:05:17,556 [info] run executed, status=completed\n",
      "final state: succeeded\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #b3edff;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #ffe6cc;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #ffe6cc;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>getting-started-iris-iguazio</td>\n",
       "      <td><div title=\"27133308b9d944ad93588934c9f34284\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/27133308b9d944ad93588934c9f34284/info\" target=\"_blank\" >...c9f34284</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 15 08:05:17</td>\n",
       "      <td>completed</td>\n",
       "      <td>serving-tester</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">host=serving-tester-t4z5v</div></td>\n",
       "      <td><div title=\"store://default/train_test_set#cea10bf1c2bd440f9da605ec3c804dde\">table</div></td>\n",
       "      <td><div class=\"dictlist\">addr=http://192.168.224.70:31378</div><div class=\"dictlist\">model=iris_dataset_v1</div></td>\n",
       "      <td><div class=\"dictlist\">total_tests=15</div><div class=\"dictlist\">errors=0</div><div class=\"dictlist\">match=14</div><div class=\"dictlist\">avg_latency=9715</div><div class=\"dictlist\">min_latency=9294</div><div class=\"dictlist\">max_latency=10622</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result86b72c07\" title=\"/files/getting-started-tutorial/jobs/latency.html\">latency</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result86b72c07-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result86b72c07-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result86b72c07\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result86b72c07-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to track results use .show() or .logs() or in CLI: \n",
      "!mlrun get run 27133308b9d944ad93588934c9f34284 --project getting-started-iris-iguazio , !mlrun logs 27133308b9d944ad93588934c9f34284 --project getting-started-iris-iguazio\n",
      "> 2020-10-15 08:05:20,994 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "tester = serving_tester.run(name='serving-tester',\n",
    "                            params={'addr': serving_addr,\n",
    "                                    'model': model_name},\n",
    "                            inputs={'table': train_run.outputs['test_set']},\n",
    "                            artifact_path=artifact_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-test-model-via-ui\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the Model from the Dashboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also run the model-serving function from the MLRun dashboard.\n",
    "The function name is a concatenation of the project name and `-model-server`.\n",
    "Use the following code to obtain the function name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'getting-started-iris-iguazio-model-server'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "serving.spec.base_spec.get('metadata').get('name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Append `iris_dataset_v1/predict` to the function's URL to create the `POST` URL; change the body type to `JSON`; and add the following text to the request body:\n",
    "```json\n",
    "{\"instances\": [[5.1, 3.5, 1.4, 0.2]]}\n",
    "```\n",
    "You can perform these steps from the platform dashboard:\n",
    "select **Projects** from the side navigation menu; select your project (\"getting-started-iris-iguazio\"); and then select the `getting-stated-iris-iguazio-sklearn-server` function and execute the required steps, as demonstrated in the following dashboard flow:\n",
    "<br><br>\n",
    "![getting_started-pipeline](./assets/getting_started-model_serving_test.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-step-create-n-run-ml-pipeline\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Create and Run a Fully Automated ML Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You're now ready to create a full ML pipeline.\n",
    "This is done by using [Kubeflow Pipelines](https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/), which is integrated into the Iguazio Data Science Platform.\n",
    "Kubeflow Pipelines is an open-source framework for building and deploying portable, scalable machine-learning workflows based on Docker containers.\n",
    "MLRun leverages this framework to take your existing code and deploy it as steps in the pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-pipeline-workflow-define-n-save\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define and Save a Pipeline Workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A pipeline is created by running an MLRun **\"workflow\"**.\n",
    "The following code defines a workflow and writes it to a file in your project directory (**/User/getting-started-tutorial/jobs/workflow.py**).\n",
    "The workflow describes a directed acyclic graph (DAG) for execution using Kubeflow Pipelines, and depicts the connections between the functions and the data as part of an end-to-end pipeline.\n",
    "The workflow file has two parts &mdash; initialization of the function objects, and definition of a pipeline DSL (domain-specific language) for connecting the function inputs and outputs.\n",
    "Examine the code to see how functions objects are initialized and used (by name) within the workflow.\n",
    "\n",
    "The defined pipeline includes the following steps:\n",
    "\n",
    "- Ingest the Iris flower data set (`ingest`).\n",
    "- Analyze the data set (`describe`).\n",
    "- Train and test the model with hyperparameters (`train`).\n",
    "- Deploy the model as a real-time serverless function (`deploy`).\n",
    "- Test the serverless serving-model function with a test data set by using REST API calls (`Tester`).\n",
    "\n",
    "> **Note**: A pipeline can also include continuous build integration and deployment (CI/CD) steps, such as building container images and deploying models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /User/getting-started-tutorial/conf/workflow.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {path.join(project_path, 'workflow.py')}\n",
    "\n",
    "from kfp import dsl\n",
    "from mlrun import mount_v3io\n",
    "\n",
    "funcs = {}\n",
    "DATASET = 'iris_dataset'\n",
    "LABELS = \"label\"\n",
    "MODELS = [\"sklearn.ensemble.RandomForestClassifier\",\n",
    "          \"sklearn.linear_model.LogisticRegression\",\n",
    "          \"sklearn.ensemble.AdaBoostClassifier\"]\n",
    "\n",
    "\n",
    "# Configure function resources and local settings\n",
    "def init_functions(functions: dict, project=None, secrets=None):\n",
    "    for f in functions.values():\n",
    "        f.apply(mount_v3io())\n",
    "\n",
    "    functions['serving'].metadata.name = 'getting-started-serving'\n",
    "\n",
    "\n",
    "# Create a Kubeflow Pipelines pipeline\n",
    "@dsl.pipeline(\n",
    "    name=\"Getting-started tutorial\",\n",
    "    description=\"This tutorial is designed to demonstrate some of the main \"\n",
    "                \"capabilities of the Iguazio Data Science Platform.\\n\"\n",
    "                \"The tutorial uses the Iris flower data set.\"\n",
    ")\n",
    "def kfpipeline(source_url='https://s3.wasabisys.com/iguazio/data/iris/iris_dataset.csv'):\n",
    "\n",
    "    # Ingest the data set\n",
    "    ingest = funcs['get-data'].as_step(\n",
    "        name=\"get-data\",\n",
    "        handler='get_data',\n",
    "        inputs={'source_url': source_url},\n",
    "        params={'format': 'pq'},\n",
    "        outputs=[DATASET])\n",
    "\n",
    "    # Analyze the dataset\n",
    "    describe = funcs[\"describe\"].as_step(\n",
    "        name=\"summary\",\n",
    "        params={\"label_column\": LABELS},\n",
    "        inputs={\"table\": ingest.outputs[DATASET]})\n",
    "\n",
    "    # Train a model with hyperparemeters\n",
    "    train = funcs[\"train\"].as_step(\n",
    "        name=\"train\",\n",
    "        params={\"sample\": -1,\n",
    "                \"label_column\": LABELS,\n",
    "                \"test_size\": 0.10},\n",
    "        hyperparams={'model_pkg_class': MODELS},\n",
    "        selector='max.accuracy',\n",
    "        inputs={\"dataset\": ingest.outputs[DATASET]},\n",
    "        outputs=['model', 'test_set'])\n",
    "\n",
    "    # Test and visualize the model\n",
    "    test = funcs[\"test\"].as_step(\n",
    "        name=\"test\",\n",
    "        params={\"label_column\": LABELS},\n",
    "        inputs={\"models_path\": train.outputs['model'],\n",
    "                \"test_set\": train.outputs['test_set']})\n",
    "\n",
    "    # Deploy the model as a serverless function\n",
    "    deploy = funcs[\"serving\"].deploy_step(\n",
    "        models={f\"{DATASET}_v1\": train.outputs['model']})\n",
    "\n",
    "    # Test the new model server (via REST API calls)\n",
    "    tester = funcs[\"serving-tester\"].as_step(\n",
    "        name='serving-tester',\n",
    "        params={'addr': deploy.outputs['endpoint'], 'model': f\"{DATASET}_v1\"},\n",
    "        inputs={'table': train.outputs['test_set']})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Register the Workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `set_workflow` MLRun project method to register your workflow with MLRun.\n",
    "The following code sets the `name` parameter to the selected workflow name (\"main\") and the `code` parameter to the name of the workflow file that is found in your project directory (**workflow.py**)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register the workflow file as \"main\"\n",
    "project.set_workflow('main', 'workflow.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-save-project\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Your Project Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `save` MLRun project method to save your project definitions to a project-configuration file in your project directory (**/User/getting-started-tutorial/conf**).\n",
    "The default name of the project file is **project.yaml**, but you can optionally change it by setting the `filepath` parameter of the `save` method.\n",
    "\n",
    "> **Note:** It's recommended that you commit your project-configuration file and any future changes to this file to a Git repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "project.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='gs-pipeline-workflow-run'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Your Pipeline Workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `run` MLRun project method to execute your workflow pipeline with Kubeflow Pipelines.\n",
    "The tutorial code sets the following method parameters; (for the full parameters list, see the MLRun documentation or embedded help):\n",
    "\n",
    "- **`name`** &mdash; the workflow name (in this case, \"main\" &mdash; see the previous step).\n",
    "- **`arguments`** &mdash; A dictionary of Kubeflow Pipelines arguments (parameters).\n",
    "  The tutorial code sets this parameter to an empty arguments list (`{}`), but you can edit the code to add arguments.\n",
    "- **`artifact_path`** &mdash; a path or URL that identifies a location for storing the workflow artifacts.\n",
    "  You can use `{{workflow.uid}}` in the path to signify the ID of the current workflow run iteration.\n",
    "  The tutorial code sets the artifacts path to a **&lt;worker ID&gt;** directory (`{{workflow.uid}}`) in a **pipeline** directory within the current tutorial directory (**/User/getting-started-tutorial/pipeline/&lt;worker ID&gt;**).\n",
    "- **`dirty`** &mdash; set to `True` to allow running the workflow also when the project's Git repository is dirty (i.e., contains uncommitted changes).\n",
    "  (When the notebook that contains the execution code is in the same Git directory as the executed workflow, the directory will always be dirty during the execution.)\n",
    "\n",
    "The `run` method returns the ID of the executed workflow, which the code stores in a `run_id` variable.\n",
    "You can use this ID to track the progress or your workflow, as demonstrated in the following sections.\n",
    "\n",
    "> **Note**: You can also run the workflow from a command-line shell by using the `mlrun` CLI.\n",
    "> The following CLI command defines a similar execution logic as that of the `run` call in the tutorial:\n",
    "> ```\n",
    "> mlrun project /User/getting-started-tutorial/conf -r main -p \"$V3IO_HOME_URL/getting-started-tutorial/pipeline/{{workflow.uid}}/\"\n",
    "> ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2020-10-15 08:05:21,213 [info] using in-cluster config.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Experiment link <a href=\"https://dashboard.default-tenant.app.dev34.lab.iguazeng.com/pipelines/#/experiments/details/ba702057-3aaa-4f38-9522-6901087caec6\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run link <a href=\"https://dashboard.default-tenant.app.dev34.lab.iguazeng.com/pipelines/#/runs/details/a8372d06-a416-4b18-8e02-8260ab9c5d68\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2020-10-15 08:05:21,922 [info] Pipeline run id=a8372d06-a416-4b18-8e02-8260ab9c5d68, check UI or DB for progress\n",
      "> 2020-10-15 08:05:21,923 [info] waiting for pipeline run completion\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<h2>Run Results</h2>Workflow a8372d06-a416-4b18-8e02-8260ab9c5d68 finished, status=Succeeded<br>click the hyper links below to see detailed results<br><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>uid</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td><div title=\"807e1c4790ce4a36856a1b7d0880ab82\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/807e1c4790ce4a36856a1b7d0880ab82/info\" target=\"_blank\" >...0880ab82</a></div></td>\n",
       "      <td>Oct 15 08:06:29</td>\n",
       "      <td>completed</td>\n",
       "      <td>serving-tester</td>\n",
       "      <td><div class=\"dictlist\">total_tests=15</div><div class=\"dictlist\">errors=0</div><div class=\"dictlist\">match=14</div><div class=\"dictlist\">avg_latency=21718</div><div class=\"dictlist\">min_latency=21059</div><div class=\"dictlist\">max_latency=24056</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/latency.html\">latency</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"9cd49a1221d5400ba1d7ad82eec7451a\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/9cd49a1221d5400ba1d7ad82eec7451a/info\" target=\"_blank\" >...eec7451a</a></div></td>\n",
       "      <td>Oct 15 08:06:13</td>\n",
       "      <td>completed</td>\n",
       "      <td>test</td>\n",
       "      <td><div class=\"dictlist\">accuracy=0.9333333333333333</div><div class=\"dictlist\">test-error=0.06666666666666667</div><div class=\"dictlist\">auc-micro=0.9311111111111112</div><div class=\"dictlist\">auc-weighted=0.955982905982906</div><div class=\"dictlist\">f1-score=0.9137254901960784</div><div class=\"dictlist\">precision_score=0.8888888888888888</div><div class=\"dictlist\">recall_score=0.9629629629629629</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/feature-importances.html\">feature-importances</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/precision-recall-multiclass.html\">precision-recall-multiclass</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/roc-multiclass.html\">roc-multiclass</div><div title=\"/User/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/test_set_preds.parquet\">test_set_preds</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"f931239128f24a66ad92f3a619d38de4\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/f931239128f24a66ad92f3a619d38de4/info\" target=\"_blank\" >...19d38de4</a></div></td>\n",
       "      <td>Oct 15 08:05:51</td>\n",
       "      <td>completed</td>\n",
       "      <td>summary</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/hist.html\">histograms</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/violin.html\">violin</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/imbalance.html\">imbalance</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/imbalance-weights-vec.csv\">imbalance-weights-vec</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/correlation-matrix.csv\">correlation-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/corr.html\">correlation</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"712ec7a7a05740d68467549d737dca8e\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/712ec7a7a05740d68467549d737dca8e/info\" target=\"_blank\" >...737dca8e</a></div></td>\n",
       "      <td>Oct 15 08:05:50</td>\n",
       "      <td>completed</td>\n",
       "      <td>train</td>\n",
       "      <td><div class=\"dictlist\">best_iteration=1</div><div class=\"dictlist\">accuracy=0.9705882352941176</div><div class=\"dictlist\">test-error=0.029411764705882353</div><div class=\"dictlist\">auc-micro=0.9969723183391004</div><div class=\"dictlist\">auc-weighted=0.9949732620320856</div><div class=\"dictlist\">f1-score=0.9679633867276888</div><div class=\"dictlist\">precision_score=0.9666666666666667</div><div class=\"dictlist\">recall_score=0.9722222222222222</div></td>\n",
       "      <td><div title=\"/User/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/data/1/test_set.parquet\">test_set</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/model/plots/1/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/model/plots/1/feature-importances.html\">feature-importances</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/model/plots/1/precision-recall-multiclass.html\">precision-recall-multiclass</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/model/plots/1/roc-multiclass.html\">roc-multiclass</div><div title=\"/User/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/model/1\">model</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/iteration_results.csv\">iteration_results</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"f27341ad483142678bf99d8212254b57\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/f27341ad483142678bf99d8212254b57/info\" target=\"_blank\" >...12254b57</a></div></td>\n",
       "      <td>Oct 15 08:05:34</td>\n",
       "      <td>completed</td>\n",
       "      <td>get-data</td>\n",
       "      <td></td>\n",
       "      <td><div title=\"/User/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/data/iris_dataset.parquet\">iris_dataset</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "run_id = project.run(\n",
    "    'main',\n",
    "    arguments={}, \n",
    "    artifact_path=path.abspath(path.join('pipeline','{{workflow.uid}}')), \n",
    "    dirty=True,\n",
    "    watch=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-pipeline-workflow-run-output-review\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review the Run Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The run output contains hyperlinks to the relevant sections for your workflow in the platform dashboard's pipelines dashboard (**Pipelines**):\n",
    "\n",
    "Select the **Experiments link** to view information about the executed experiments.\n",
    "<br><br>\n",
    "![getting_started-pipeline](./assets/getting_started-pipeline-experiment.png)\n",
    "\n",
    "Select the **Run link** to view the pipeline execution.\n",
    "The following image illustrates the execution graph for a completed run:\n",
    "<br><br>\n",
    "![getting_started-pipeline](./assets/getting_started-pipeline.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-pipeline-workflow-track-execution\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Track the Pipeline Workflow Execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the following code to view the information for your pipeline workflow execution &mdash; including inputs and outputs, run results, and artifacts.\n",
    "\n",
    "The code uses the `get_run_db` MLRun method to get an MLRun DB object for your project's MLRun database/API service (\"the MLRun DB\"), and the `connect` method to connect to the DB.\n",
    "The returned database object is stored in a `db` variable, which can then be used to run additional methods for performing different operations on the database.\n",
    "\n",
    "The code uses the `list_runs` method to list all runs for the current workflow.\n",
    "The workflow is referenced by using the workflow ID that was returned for the workflow execution and saved to the `run_id` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #b3edff;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #ffe6cc;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #ffe6cc;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>getting-started-iris-iguazio</td>\n",
       "      <td><div title=\"807e1c4790ce4a36856a1b7d0880ab82\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/807e1c4790ce4a36856a1b7d0880ab82/info\" target=\"_blank\" >...0880ab82</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 15 08:06:29</td>\n",
       "      <td>completed</td>\n",
       "      <td>serving-tester</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">workflow=a8372d06-a416-4b18-8e02-8260ab9c5d68</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">host=serving-tester-lwl56</div></td>\n",
       "      <td><div title=\"store://getting-started-iris-iguazio/train_test_set#a8372d06-a416-4b18-8e02-8260ab9c5d68\">table</div></td>\n",
       "      <td><div class=\"dictlist\">addr=http://192.168.224.70:31012</div><div class=\"dictlist\">model=iris_dataset_v1</div></td>\n",
       "      <td><div class=\"dictlist\">total_tests=15</div><div class=\"dictlist\">errors=0</div><div class=\"dictlist\">match=14</div><div class=\"dictlist\">avg_latency=21718</div><div class=\"dictlist\">min_latency=21059</div><div class=\"dictlist\">max_latency=24056</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/latency.html\">latency</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>getting-started-iris-iguazio</td>\n",
       "      <td><div title=\"9cd49a1221d5400ba1d7ad82eec7451a\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/9cd49a1221d5400ba1d7ad82eec7451a/info\" target=\"_blank\" >...eec7451a</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 15 08:06:13</td>\n",
       "      <td>completed</td>\n",
       "      <td>test</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">workflow=a8372d06-a416-4b18-8e02-8260ab9c5d68</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">host=test-mkgrb</div></td>\n",
       "      <td><div title=\"store://getting-started-iris-iguazio/train_model#a8372d06-a416-4b18-8e02-8260ab9c5d68\">models_path</div><div title=\"store://getting-started-iris-iguazio/train_test_set#a8372d06-a416-4b18-8e02-8260ab9c5d68\">test_set</div></td>\n",
       "      <td><div class=\"dictlist\">label_column=label</div></td>\n",
       "      <td><div class=\"dictlist\">accuracy=0.9333333333333333</div><div class=\"dictlist\">test-error=0.06666666666666667</div><div class=\"dictlist\">auc-micro=0.9311111111111112</div><div class=\"dictlist\">auc-weighted=0.955982905982906</div><div class=\"dictlist\">f1-score=0.9137254901960784</div><div class=\"dictlist\">precision_score=0.8888888888888888</div><div class=\"dictlist\">recall_score=0.9629629629629629</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/feature-importances.html\">feature-importances</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/precision-recall-multiclass.html\">precision-recall-multiclass</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/roc-multiclass.html\">roc-multiclass</div><div title=\"/User/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/test_set_preds.parquet\">test_set_preds</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>getting-started-iris-iguazio</td>\n",
       "      <td><div title=\"f931239128f24a66ad92f3a619d38de4\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/f931239128f24a66ad92f3a619d38de4/info\" target=\"_blank\" >...19d38de4</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 15 08:05:51</td>\n",
       "      <td>completed</td>\n",
       "      <td>summary</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">workflow=a8372d06-a416-4b18-8e02-8260ab9c5d68</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">host=summary-72tmj</div></td>\n",
       "      <td><div title=\"store://getting-started-iris-iguazio/get-data_iris_dataset#a8372d06-a416-4b18-8e02-8260ab9c5d68\">table</div></td>\n",
       "      <td><div class=\"dictlist\">label_column=label</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/hist.html\">histograms</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/violin.html\">violin</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/imbalance.html\">imbalance</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/imbalance-weights-vec.csv\">imbalance-weights-vec</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/correlation-matrix.csv\">correlation-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/plots/corr.html\">correlation</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>getting-started-iris-iguazio</td>\n",
       "      <td><div title=\"712ec7a7a05740d68467549d737dca8e\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/712ec7a7a05740d68467549d737dca8e/info\" target=\"_blank\" >...737dca8e</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 15 08:05:50</td>\n",
       "      <td>completed</td>\n",
       "      <td>train</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">workflow=a8372d06-a416-4b18-8e02-8260ab9c5d68</div><div class=\"dictlist\">kind=job</div></td>\n",
       "      <td><div title=\"store://getting-started-iris-iguazio/get-data_iris_dataset#a8372d06-a416-4b18-8e02-8260ab9c5d68\">dataset</div></td>\n",
       "      <td><div class=\"dictlist\">sample=-1</div><div class=\"dictlist\">label_column=label</div><div class=\"dictlist\">test_size=0.1</div></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=1</div><div class=\"dictlist\">accuracy=0.9705882352941176</div><div class=\"dictlist\">test-error=0.029411764705882353</div><div class=\"dictlist\">auc-micro=0.9969723183391004</div><div class=\"dictlist\">auc-weighted=0.9949732620320856</div><div class=\"dictlist\">f1-score=0.9679633867276888</div><div class=\"dictlist\">precision_score=0.9666666666666667</div><div class=\"dictlist\">recall_score=0.9722222222222222</div></td>\n",
       "      <td><div title=\"/User/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/data/1/test_set.parquet\">test_set</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/model/plots/1/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/model/plots/1/feature-importances.html\">feature-importances</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/model/plots/1/precision-recall-multiclass.html\">precision-recall-multiclass</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/model/plots/1/roc-multiclass.html\">roc-multiclass</div><div title=\"/User/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/model/1\">model</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resulte14510b9\" title=\"/files/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/iteration_results.csv\">iteration_results</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>getting-started-iris-iguazio</td>\n",
       "      <td><div title=\"f27341ad483142678bf99d8212254b57\"><a href=\"https://mlrun-ui.default-tenant.app.dev34.lab.iguazeng.com/projects/getting-started-iris-iguazio/jobs/f27341ad483142678bf99d8212254b57/info\" target=\"_blank\" >...12254b57</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 15 08:05:34</td>\n",
       "      <td>completed</td>\n",
       "      <td>get-data</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=iguazio</div><div class=\"dictlist\">owner=iguazio</div><div class=\"dictlist\">workflow=a8372d06-a416-4b18-8e02-8260ab9c5d68</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">host=get-data-hskx4</div></td>\n",
       "      <td><div title=\"https://s3.wasabisys.com/iguazio/data/iris/iris_dataset.csv\">source_url</div></td>\n",
       "      <td><div class=\"dictlist\">format=pq</div></td>\n",
       "      <td></td>\n",
       "      <td><div title=\"/User/getting-started-tutorial/pipeline/a8372d06-a416-4b18-8e02-8260ab9c5d68/data/iris_dataset.parquet\">iris_dataset</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resulte14510b9-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resulte14510b9-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resulte14510b9\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resulte14510b9-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from mlrun import get_run_db\n",
    "db = get_run_db().connect()\n",
    "db.list_runs(project=project.name, labels=f'workflow={run_id}').show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-done\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Done!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulation. You've completed your first Iguazio Data Science Platform tutorial!.\n",
    "\n",
    "- To learn more about deploying live endpoints and concept drift, refer to the **network-operations (NetOps) demo**:\n",
    "    <table width=\"100%\" align=\"left\" align=\"left\">\n",
    "        <tr>\n",
    "            <td width=\"10%\">\n",
    "                <a href=\"demos/network-operations/project.ipynb\"><img src=\"../assets/images/Jupyter-Logo-32px.png\" /><br>Open locally</a>\n",
    "            </td>\n",
    "            <td>\n",
    "                <a target=\"_blank\" href=\"https://github.com/mlrun/demos/tree/master/network-operations/\"><img src=\"../assets/images/GitHub-Mark-32px.png\" /><br>View on GitHub</a>\n",
    "            </td>\n",
    "        </tr>\n",
    "    </table>\n",
    "\n",
    "- For an example of distributed training of an image-classification pipeline using TensorFlow (versions 1 or 2), Keras, and Horovod, refer to the **image-classification with distributed training demo**:\n",
    "    <table width=\"100%\" align=\"left\" align=\"left\">\n",
    "        <tr>\n",
    "            <td width=\"10%\">\n",
    "                <a href=\"demos/image-classification-with-distributed-training/horovod-project.ipynb\"><img src=\"../assets/images/Jupyter-Logo-32px.png\" /><br>Open locally</a>\n",
    "            </td>\n",
    "            <td>\n",
    "                <a target=\"_blank\" href=\"https://github.com/mlrun/demos/tree/master/image-classification-with-distributed-training/\">\n",
    "                    <img src=\"../assets/images/GitHub-Mark-32px.png\" /><br>View on GitHub</a>\n",
    "            </td>\n",
    "        </tr>\n",
    "    </table>\n",
    "- For more information about these demos and additional demos, refer to the [**welcome.ipynb**](welcome.ipynb#end-to-end-use-case-applications) notebook or matching [**README.md**](README.md#end-to-end-use-case-applications) file.\n",
    "- To learn about ingesting data into the platform and engineering the data using different technologies &mdash; such as Spark, Dask, SQL queries and RAPIDS &mdash; refer to the **data ingestion and preparation overview** &mdash; [notebook](../data-ingestion-and-preparation/README.ipynb) / [Markdown](../data-ingestion-and-preparation/README.md)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
